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ABSTRACT 


This  research  effort  has  as  its  principal  objective,  the  collection  of  data  to  determine  if 
there  is  a  relationship  between  the  number  of  Engineering  and  Manufacturing 
Development  (EMD)  test  articles  used  and  the  managerial  success  of  that  program  within 
the  EMD  phase  of  a  major  system  acquisition.  Managerial  success  is  largely  measured  in 
terms  of  a  low  schedule  overrun  during  EMD.  The  research  approach  taken  also  gathered 
other  acquisition  related  data  discussed  herein.  This  allowed  comment  on  the  current 
“track  record”  of  DoD  cost  and  schedule  overruns  during  EMD,  and  some  interesting  data 
on  six  other  variables  that  could  affect  program  success.  The  conclusion  is  that  there  is  a 
positive  correlation  between  the  number  of  test  articles  and  program  success.  However, 
these  conclusions  may  have  been  impacted  by  the  nuances  of  the  DoD  Low  Rate  Initial 
Production  (LRIP)  process  and  the  basic  design  of  this  study. 
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IN  THE  ACQUISITION 
PROCESS 


CHAPTER  1 
INTRODUCTION 


Background 

For  major  acquisition  programs  which  include  a  Low  Rate  Initial  Production  (LRIP) 
phase,  the  Director  of  Operational  Test  and  Evaluation  (DOT&E)  is  responsible  for 
approving  the  number  of  LRIP  test  articles  required  for  Initial  Operational  Test  and 
Evaluation  (lOT&E).'  Since  1991,  Congressional  Law  requires  the  DOT&E  to  specify  at 
Milestone  II,  the  number  of  test  articles  required  for  this  test.^  The  determination  of  LRIP 
test  quantities  is  a  difficult  trade-off  among  several  factors,  and  the  office  of  the  DOT&E 
desired  to  know  if  there  was  historical  data  that  would  help  in  making  this  important 
decision.  Therefore,  they  requested  the  Defense  Systems  Management  College  (DSMC)  to 
study  if  there  is  a  relationship  between  the  number  of  test  articles  used  in  Engineering  and 
Manufacturing  Development  (EMD),  including  lOT&E,  and  the  success  of  that  program. 
Indications  were  factual  references  or  metrics  relating  to  the  subject  were  non-existent.^ 

Overview 

The  research  approach  selected  was  an  ex  post  facto  casual-comparative  effort  whose 
hypothesis  was: 

The  success  of  a  weapon  system  development  program  is  directly 

related  to  the  Initial  Production  Test  Articles  used  in  the  system  EMD  Phase, 

including  the  lOT&E  test. 

An  immediate  concern  was  for  defining  the  terms  to  be  used  within  the  study.  This  was 
necessary  because  many  of  the  parameters  vital  to  the  study  were  known  to  have  differing 
interpretations  throughout  the  Department  of  Defense  (DoD)  acquisition  organization. 
Appendix  A  lists  the  definitions  used  within  this  study. 

For  convenience,  the  term  LRIP  is  used  throughout  this  study,  and  is  synonymous  with 
the  term  Initial  Production  Articles,  except  for  the  type  of  funds  used  to  procure  the  test 
articles.  In  this  study  the  term  “LRIP  test  quantity”  is  used  to  reflect  all  systems  procured 
during  EMD  with  Research,  Development,  Test  and  Evaluation  (RDT&E)  funds.  This  is 
the  quantity  of  Initial  Production  Articles,  less  those  bought  with  procurement  funds.  The 
reason  for  this  distinction  is  the  presumption  the  RDT&E  funded  items  are  those  used  for 
testing.  Chapter  6  contains  details  regarding  this  distinction. 

The  LRIP  phase  of  a  program  occurs  during  the  EMD  phase  of  a  program;  that  is, 
between  Milestones  (M/S)  II  and  III,  (see  Figure  1).  The  objectives  of  the  LRIP  phase 
are  specified  by  statute  to  be: 

....the  production  of  the  system  in  the  minimum  quantity  necessary- 

(1)  to  provide  production-configured  or  representative  articles  for  operation- 
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al  tests  pursuant  to  section  2399  of  this  title;  (2)  to  establish  an  initial  production 
base  for  the  system;  and  (3)  to  permit  an  orderly  increase  in  the  production 
rate  for  the  system  sufficient  to  lead  to  full-rate  production  upon  the  successful 
completion  of  operational  testingf 

All  data  used  within  this  report  are  unclassified. 

It  is  important  to  recognize  that  our  evaluation  scheme  is  a  measure  of  program 
management  success  in  terms  of  schedule  slip  rather  than  eventual  weapon  system  success. 
There  was  no  attempt  to  survey  the  effectiveness  of  the  systems  in  their  operational  roles. 
Also,  we  did  not  evaluate  the  performance  (effectiveness  and  suitability)  of  these  programs 
in  EMD  although  that  would  be  an  excellent  additional  study. 

The  subsequent  chapters  of  this  report  are  organized  as  follows:  Chapter  2  discusses 
the  scope  of  the  literature  search  conducted,  and  indicates  most  prior  research  in  this 
general  area  supports  the  findings  of  this  research.  Chapter  3  discusses  the  approach 
taken  and  describes  the  fortuitous  opportunity  presented  to  conduct  a  pilot  study.  Chapter 
4  describes  the  limitations  and  assumptions  implicit  in  this  research.  Chapter  5  is  an 
analysis  of  this  effort,  limited  to  the  degree  possible  to  facts  that  emerge  from  the 
database.  Chapter  6  is  a  discussion  of  opinions  resulting  from  one  viewpoint  of  the  facts 
contained  in  the  data  bank.  Considerable  important  data  is  also  contained  within  the 
remaining  areas  of  this  report. 
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CHAPTER  2 

REVIEW  OF  RELATED  LITERATURE 


Discussion 

The  primary  sources  used  for  this  report  were  the  Defense  Technical  Information  Center 
(DTIC),  Research  on  Ongoing  Acquisition  Research  (ROAR),  ^  the  library  research 
database,  and  the  Industry  Field  Trip  database.®  We  also  inquired  by  corporate  author, 
and  reviewed  the  master’s  thesis  written  at  the  Naval  Postgraduate  School.  The  RAND 
Corporation  reports  were  also  veiy  helpful.  Of  course,  the  Department  of  Defense  (DoD) 
Directives  and  other  Government  guidance  documents  were  utilized.  Reports  reviewed 
are  listed  within  the  references  and  bibliography.  Some  of  the  prior  reports  most  germane 
and  useful  are  discussed  next. 

Related  References 

Drezner’s,  The  Nature  and  Role  of  Prototyping  in  Weapon  System  Development, 
provides  us  with  an  excellent  definition  of  a  prototype.  ’  Fortuitously,  his  definition  of 
prototype  is  exactly  what  an  initial  production  article,  as  used  in  this  report,  is  not. 
Therefore,  there  is  a  clean  interface  between  these  terms  frequently  used  interchangeably, 
and  therefore  erroneously.  Drezner  and  Smith’s  An  Analysis  of  Weapon  System 
Acquisition  Schedules  *  reviewed  the  time  it  took  to  develop  weapon  systems  over  the 
decades  of  the  1950’s,  60’ s,  70’s  and  80’s.  Two  of  their  findings,  namely  that  schedule 
estimates  are  not  given  the  critical  evaluation  that  cost  estimates  are  given,  and  that  there 
is  no  relationship  between  the  time  a  program  spends  in  the  DemonstrationA^alidation 
(DemA^al)  phase  and  a  reduced  schedule  slip,  agree  with  our  findings.  A  third  finding, 
removing  the  effects  of  external  factors  reduces  program  length  by  a  substantial  amount,  is 
intuitively  agreed  with.  However,  our  study  excluded  any  consideration  of  extra-program 
influences. 

Rich  and  Dews’  Improving  the  Military  Acquisition  Process,^  contains  a  finding  that  is 
exactly  opposite  to  our  own.  It  states  for  programs  in  the  1960’s,  cost  overruns 
approximated  45  percent,  schedule  overruns  15  percent  and  performance  shortfalls  5 
percent.  “These  findings  support  the  conventional  wisdom  that  when  acquisition  problems 
arise  cost  is  the  constant  most  easily  relaxed  and  schedule  is  next,  whereas,  performance 
goals  are  adhered  to  most  closely.” 

Such  was  the  case  in  the  1960’s.  However,  in  the  1980’s,  affordability  was  added  to 
the  criteria  for  evaluating  program  success,  along  with  cost,  schedule  and  performance. 
Most  recently,  program  requirements  generation,  which  affects  performance,  is  under 
review.  Our  own  research  on  more  recent  programs  indicates  more  programs  were  better 
able  to  control  their  cost  overruns  than  their  schedule  overruns.  This  reversal  of  priorities 
between  cost  importance  and  schedule  importance  is  considered  an  important  comparative 
research  finding. 
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Drezner,  et  al.,*®  in  a  1993  study  of  128  weapon  system  programs  that  were  three  or 
more  years  into  the  Engineering  and  Manufacturing  Development  (EMD)  phase 
concluded,  “The  weighted  average  cost  growth  of  Development  Estimate  baseline 
programs,  (Milestone  (M/S)  II) ...  as  of  December  1990  is  20  percent.”  Our  own  study 
started  at  the  same  point  but  continued  through  all  of  EMD  concluding  with  M/S  III 
actual  cost  data.  When  you  consider  that  the  average  duration  of  the  EMD  phase  for 
major  programs  is  7.4  years,"  and  a  large  degree  of  the  cost  overrun  presumably  occurs 
toward  the  end  of  EMD,  Drezner’s  figure  is  remarkably  close  to  our  own  higher  figures, 
for  the  greater  length  of  time. 

A  recent  Naval  Postgraduate  School  master’s  thesis*^  drew  the  conclusion  that  the 
majority  of  problems  which  occurred  during  lOT&E  were  directly  related  to  test  resource 
issues,  and  recommended  sufficient  test  articles  be  produced  and  available  well  before  the 
operational  test  was  supposed  to  start.  An  earlier  study  performed  by  the  Air  Force 
Operational  Test  and  Evaluation  Center  in  1991 concluded  “it  is  obvious  that  the  test 
articles  themselves  have  posed  the  greatest  proportion  of  [test]  limitations.”  Our  research 
would  support  these  earlier  independent  studies. 

Other  studies,  mentioned  briefly  here,  comment  on  requirements  generation  and  the 
program  risk  evaluation  process,  indicating  both  should  be  reviewed.  Wiles*'*  indicates 
the  requirements  generation  and  management  process  led  to  unrealistic  operational 
requirements.  Earlier,  David  Packard*^  stated  all  to  often,  requirements  for  new  weapon 
systems  have  been  overstated.  Commenting  on  the  Advanced  Medium  Range  Air  to  Air 
Missile  program’s  spotted  history,  Mayer’s*^  comments  “...the  chief  cause  of  Advanced 
Medium  Range  Air  to  Air  Missile’s  (AMRAAM’s)  woes  is  that  managers  vastly  over  sold 
the  program  in  terms  of  cost  and  schedule.” 

Glennan,  et  al.*^  reviewed  risk  evaluation  on  seven  acquisition  programs  to  determine 
how  risk  was  identified  and  managed.  He  found  that  in  three  of  the  programs  having  the 
most  difficulty,  “For  a  variety  of  reasons,  top-level  management  oversight  in  the  Air  Force 
and  DoD  discounted  lower  level  assessments  of  the  level  of  risk.”  Finally,  the  DoD 
Inspector  General**  in  a  recent  Audit  Report  found  in  seven  recent  major  acquisition 
programs,  all  entered  Low  Rate  Initial  Production  (LRIP)  prematurely  and  LRIP 
acquisition  strategies  did  not  effectively  limit  production  quantities  before  M/S  III. 

Summary  of  Literature  Reviewed 

Approximately  600  titles  were  extracted  from  the  Defense  Technical  Information 
Center  (DTIC)  and  other  Acker  Library  research  reference  databases.  An  estimated  100 
bibliographies  were  ordered,  and  approximately  50  reports  were  scanned  or  read  in 
support  of  this  research  effort.  With  the  exception  of  the  dis-similar  findings  discussed 
previously,  the  great  majority  of  prior  research  findings  were  either  very  closely 
supportive,  or  generally  supportive  of  the  findings  of  this  research. 
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CHAPTERS 

METHODOLOGY 


Concept  and  Approach 

An  underlying  concept  to  this  study  was  to  use  only  official  data  available  to 
Department  of  Defense  (DoD)  principals  and  staff  at  the  Defense  Acquisition  Board 
(DAB)  Milestone  (M/S)  II  and  III  meetings.  The  rationale  was  acquisition  decisions  were 
based  largely  on  this  data,  whether  it  was  complete  or  incomplete,  clear  or  hazy,  right  or 
wrong,  with  regard  to  estimates  stated.  We  therefore  limited  our  data  sources  to  the 
“Blue  Books”  used  at  DAB  M/S  meetings,  and  the  Selected  Acquisition  Report  (SAR)  for 
the  year  the  program  conducted  its  M/S  III  DAB  meeting.  Blue  Books  are  summary  plans 
and  data  compiled  for  the  use  of  DAB  principals  and  staff  assistants  immediately  prior  to 
M/S  meetings. 

The  program’s  planned  activities  and  expenditures  were  given  in  M/S  II  DAB 
information.  The  M/S  III  DAB  information  provided  us  with  actual  activities  and  costs 
incurred  during  Engineering  and  Manufacturing  Development  (EMD).  We  were 
particularly  interested  in  the  planned  and  actual  costs  for  EMD,  and  the  planned  and  actual 
schedule.  The  DAB  M/S  II  and  M/S  III  cost  data,  as  well  as  the  SAR  cost  data,  were 
adjusted  for  inflation  as  necessary  to  express  them  in  the  same  fiscal  year  dollars.*® 

Forty-one  weapon  system  programs  were  originally  considered  as  the  study  population. 
A  screening  criteria  (Appendix  B)  developed  prior  to  data  collection  produced  the  study 
sample  of  twenty-four  programs.  Table  1  depicts  the  twenty-four  programs  studied  for 
this  report. 


TABLE  1.  The  Twenty-Four  Programs  Researched 


CVHELO(SH-60F) 

ALCM 

MK  50  Torpedo  (ALWT) 

MK48  ADCAP 

Navstar  GPS/U.E. 

FAADS  LOS-F-H  (ADATS) 

Avenger  (Ped  MT  Stinger) 

AMRAAM 

ACM 

ATACMS 

B-IB  Lancer  (ALQ-161) 

OH-58D/AHIP 

TrC-39  (TRI-TAC  Switch) 

JTIDS  (Class  2  terminals) 

Trident  n  msl  (D-5) 

HARM 

ASPJ(AL0-165) 

AV-8B  (Harrier  n) 

PLS  (FHTV)  (NDD 

Maverick  AGM-65D  (HR) 

ASAT(AF) 

SGT  YORK  gun  (DIVAD) 

Tomahawk  (TASM  &  TLAM-C) 

Pilot  Study  Conducted 

The  Defense  Systems  Management  College  (DSMC)  was  requested  to  participate  in  the 
quick  reaction  study  of  electronic  systems  acquisition  cited  in  reference  14.  At  that  point 
in  time,  we  had  just  completed  the  concept  and  approach  to  be  used  in  this  study,  and  data 
collection  was  just  beginning.  Since  the  four  major  Command,  Control,  Communication 
and  Intelligence  (C^I)  systems  and  electronic  warfare  (EW)  systems  of  interest  to  Dr. 
Wiles  were  among  our  study  sample,  and  since  his  study  required  completion  in  a  short 
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ninety-days,  we  used  this  opportunity  to  pilot  the  data  collection  and  analyses  planned  for 
the  entire  twenty-four  programs.  This  pilot  effort  supported  our  planned  study  design, 
and  our  contribution  to  the  Wiles-led  study  effort  has  been  published.^” 

Database  Description 

Central  to  the  analysis  and  conclusions  of  this  study  is  the  definition  of  success.  This  is 
detailed  in  Table  2.  Only  the  cost  and  schedule  success  criteria  are  covered  in  this  study. 
Performance  success  will  be  assessed  in  a  follow-on  effort.  A  scoring  concept  was 
devised  to  classify  the  programs  and  process  the  results  as  follows; 

Score  Classification  of  Program 

5  Very  Successful 

4  Successful 

3  Fairly  Successful 

2  Marginally  Successful 

1  Not  Successful 

TABLE  2.  Definition  of  the  Success  of  a  Program 


For  purposes  of  this  research  study,  the  success  of  a  DoD  acquisition  program  will  be  categorized  in  one  of 
five  ways.  Either  the  program  is  Very  Successful,  Successful,  Fairly  Successful,  Marginally  Successful,  or 
Not  Successful.  The  subjective  description  of  these  categories  is  shown  below.  For  this  report,  only  the 
portion  of  the  definition  relating  to  cost  or  schedule  overruns  apply.  The  remainder  of  the  definitions 
apply  to  the  proposed  follow-on  effort. 


Very  Successful  (Score  of  5) 

There  are  few  if  any,  system  shortcomings.  The  M/S  n  program  budget  and  program  schedule  were 
essentially  adhered  to. 

The  DOT&E  M/S  HI  Beyond  LRIP  report  was  positive.  The  Service  lOT&E/OPEVAL  report  was 
positive,  namely  effective  and  suitable  without  caveat.  (If  not  suitable,  the  deficiencies  could  be  corrected 
without  major  impact  -  i.e.,  no  SAR  breach) 

Successful  (Score  of  4) 

The  ADM  from  M/S  II  and  MS/ni  DAB's  was  straightforward.  There  were  system  shortcomings.  The 
M/S  II  program  budget  and  schedule  were  slipped,  but  not  by  more  than  30%  in  cost  and  12  months  in 
schedule. 

The  DOT&E  M/S  III  Beyond  LRIP  report  was  positive.  The  Service  lOT&E/OPEVAL  report  was 
positive.  The  overall  evaluation  was  effective,  and  suitable  with  perhaps  a  few  marginally  suitable 
parameters. 
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TABLE  2  (CONI 


Fairly  Successful  (Score  of  3) 


The  ADM  from  the  M/S  II  and  M/S  III  DAB's  contained  problem  statements.  The  program  shortcomings 
were  listed,  a  few  could  be  critical.  The  M/S  11  program  budget  and  schedule  had  to  be  revised,  but  is 
within  45%  of  the  M/S  n  budget  and  no  more  than  18  months  behind  the  M/S  n  schedule. 

The  DOT&E  M/S  IE  Beyond  LRIP  report  contained  a  few  negative  comments.  The  Service 
lOT&E/OPEVAL  report  could  be  marginally  effective  and  marginally  suitable. 

Marainallv  Successful  (Score  of  2) 

The  ADM's  from  the  M/Sn  and  M/S  HI  DAB's  indicated  major  performance  and  suitability  problems 
existed.  The  program  probably  would  be  canceled  on  the  basis  of  performance  to  date,  but  other  external 
factors  are  being  considered.  The  M/S  11  program  budget  and  schedule  was  revised  more  than  once,  and 
is  now  up  to  60%  overrun  in  cost  and  two  years  behind  the  original  schedule. 

The  exit  criteria  of  the  M/S  II  ADM  was  not  completely  met.  An  outcome  of  the  M/S  ni  DAB  will  be  to 
delay  entry  into  full  rate  production. 

The  DOT&E  M/S  III  Beyond  LRIP  reported  marginally  effective  and/or  marginally  suitable.  The  Service 
lOT&E/OPEVAL  recommended,  at  best,  the  system  was  potentially  effective  and  potentially  suitable. 

Not  Successful  (Score  of  1) 

The  ADM  from  the  M/S  n  DAB  reluctantly  approved  the  continuation  of  the  program  into  EMD,  or  held 
the  program  in  the  Dem/Val  phase.  The  M/S  n  budget  is  over  60%  overran,  and  the  program  is  more 
than  2  years  behind  schedule.  A  DOT&E  Beyond  LRIP  report  will  say  not  effective  and  not  suitable. 

This  category  would  also  include  programs  that  have  in  fact  been  terminated. 

For  programs  that  have  not  had  their  M/S  HI  DAB  review  as  yet,  their  success  will  be  judged  on  the 
general  approach  discussed  herein,  and  on  the  available  official  documentation. 


Also  of  importance  were  other  variables  affecting  program  success,  and  for  which  data 
were  collected.  The  initial  list  of  variables  affecting  success  is  shown  at  Appendix  C,  and 
the  six  selected  for  data  collection  are  shown  in  Table  3.  Appendix  D  lists  the  schedule 
suecess  rating  and  the  cost  success  rating  for  each  program.  The  difference  in  the  cost  and 
the  success  rating  for  each  program  is  an  interesting  study  outcome.  A  taxonomy  of 
weapon  system  types  was  created  for  use  within  the  study  and  consists  of: 


AIC: 

Veh: 

Elec  -EW-A: 

Elec-CNR-A: 

Elec-CNR-G: 

Mis/Mu-A: 

Mis/Mu-G: 


Aircraft 

Vehicle 

Electronic-Electronic  Warfare- Airborne 

Electronic-Comm/Nav/Radar-Airbome 

Electronic-Comm/Nav/Radar-Ground 

Missile/Munition-Airbome 

Missile/Munition-Ground 
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The  entire  data  collected  for  the  twenty-four  programs  are  shown  in  Appendix  E. 
Forty-seven  data  elements  or  operands  are  listed  for  each  program,  along  with  their 
respective  definition. 

TABLE  3.  Other  Variables  Affecting  Program  Success 

Complexity  of  technical  arrangements: 

Degree  of  risk  identified  at  M/S  II 
Use  of  competition  during  Dem/Val  phase 
Use  of  competition  during  EMD  phase 
Complexity  of  business  arrangements: 

Type  of  contract 

Number  of  associate  contractors 

Joint  program 
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CHAPTER  4 

LIMITATIONS  OF  STUDY  AND  ASSUMPTIONS 


Limitations 

The  data  for  this  study  were  restricted  to  that  contained  within  the  Blue  Books 
prepared  for  the  Defense  Acquisition  Board  (DAB)  Milestone  (M/S)  II  and  III  meetings, 
and  the  program’s  annual  Selected  Acquisition  Report  (SAR)  for  the  year  in  which  the 
M/S  III  DAB  was  held.  This  was  an  early  deliberate  design  decision.  The  rationale  was 
we  wanted  to  use  only  the  data  available  to  the  decision  makers  at  the  time  the  decision 
was  made. 

In  some  instances,  the  data  within  the  Blue  Books  were  incomplete  and  in  some  cases 
the  type  of  data  contained  within  the  Blue  Books  varied  between  programs.  This  is  not 
considered  unusual  given  the  time  elapsed  since  some  of  the  programs  had  their  DAB 
meetings  and  the  dynamics  of  the  process. 

Performance  (effectiveness  and  suitability)  was  not  evaluated.  The  primary  source 
document  for  this  parameter  would  not  be  the  Blue  Books,  but  rather  the  Service  Initial 
Operational  Test  and  Evaluation  (lOT&E)  report,  and/or  the  DOT&E  “Beyond  [Low 
Rate  Initial  Production]  LRIP”  report.  These  reports  for  the  twenty-four  programs 
reviewed  could  be  added  to  the  data  as  a  follow-on  task. 

By  definition,  this  study  concerns  itself  with  the  Engineering  and  Manufacturing 
Development  (EMD)  phase  exclusively,  since  that  is  where  LRIP  resides. 

External  influences  on  the  program  were  not  considered.  It  is  recognized  that 
Congressional  requirements  and  political  and  other  interests  can  and  do  have  a  significant 
effect  on  program  success.  This  is  particularly  true  during  the  EMD  phase,  but  would  not 
be  primarily  recorded  in  the  Blue  Books. 

Program  concurrency  -  the  overlapping  of  program  development  and  program 
production  phases  -  is  a  topic  of  current  concern.  This  study  generated  little  data  that  is 
relevant  to  concurrency,  although  again,  program  concurrency  intuitively  would  have  a 
major  impact  on  program  EMD  success. 

The  degree  to  which  software  must  be  developed  for  a  system  is  another  variable  which 
would  affect  program  success.  We  had  planned  to  include  this  as  a  variable,  but 
discovered  that  the  Blue  Books  contained  very  little  data  in  this  regard. 

In  general,  the  level  of  detail  regarding  programs  was  limited  to  that  contained  within 
the  Blue  Books.  For  example,  we  could  determine  the  number  of  LRIP  articles  procured 
with  Research,  Development,  Test  and  Evaluation  (RDT&E)  funds,  and  presumably  used 
for  all  types  of  testing  within  EMD.  We  could  not,  however,  determine  the  number  of 
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LRIP  articles  used  specifically  for  the  lOT&E  test.  Even  if  we  could,  the  Department  of 
Defense  (DoD)  Directives  require  the  lOT&E  test  article  to  be  production  representative. 
The  data  did  not  allow  us  to  determine  how  production  representative  the  test  articles 
actually  were. 

Assumptions 

The  validity  of  this  study  depended  greatly  on  how  important  the  data  contained  within 
the  Blue  Books  was  to  the  actual  decisions  made  at  DAB  meetings.  It  was  assumed  the 
Blue  Book  process  evolved  and  presented,  in  executive  summary  form,  information 
central  and  vital  to  the  decision  process. 

Another  major  tenet  that  affected  the  results  of  this  study  was  test  articles  funded  by 
RDT&E.  “Development  and  preproduction  prototypes  (RDT«&E  financed)  will  be  used 
for  IOT&E.”^‘  The  Blue  Book  and  SAR  allowed  us  to  determine  the  number  of  LRIP 
units  that  were  RDT&E  funded,  and  this  number  became  the  number  used  within  this 
study. 

Lastly,  it  was  assumed  that  program  managers  and  other  acquisition  executives  adhered 
to  the  DoD  Directives  governing  LRIP  and  other  aspects  of  systems  acquisition. 
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TABLE  4.  Results/Findings/Analysis 


A.  LRIP 

1 .  The  more  LRIP  lest  quantity  articles  in  a  program,  as  a  %  of  total  production,  the  less  schedule  slippage  there  is 
inEMD.  (Col  6,  Col  20). 

2.  Programs  that  use  more  than  3%  LRIP  test  quantity  articles,  as  a  %  of  total  production,  tend  to  be  more 
successful  in  controlling  schedule  overruns.  (Col  13,  Col  20). 

3.  An  average  of  1.8%  of  the  total  planned  production  quantity  was  acquired  by  RDT&E  funds  and  presumably  used 
for  testing  during  EMD.  (Col  16,  Col  19). 

4.  An  average  of  28%  of  the  total  LRIP  quantity  was  acquired  with  RDT&E  funds  and  presumably  used  for  testing 
during  EMD.  (Col  16,  Col  18). 

B.  GENERAL 

5.  The  acquisition  history  of  24  DoD  programs  show  an  average  cost  overrun  in  the  EMD  phase  of  the  programs 
to  be  45%,  and  the  schedule  overrun  to  be  63%.  (Col  6,  Col  10). 

6.  The  average  cost  overruns  of  18  programs  was  20%  and  the  average  schedule  overrun  was  32%.  (Cols  6, 10). 

7.  Comparisons: 

#  of  Programs  Cost  0/R  Schedule  0/R 

24  45%  63% 

18  20%  32% 

8.  In  most  weapon  systems  categories,  (except  A/C),  the  time  overrun  was  greater  than,  or  equal  to,  the  cost 
overrun.  (Col  6,  Col  10). 

C.  OTHER  VARIABLES 

9.  Programs  that  used  competition  in  Dem/Val  had  lower  Schedule  Program  Success  than  programs  that  did  not 
use  competition.  (Col  13,  Col  3 1). 

10.  The  use  of  competition  during  EMD  decreases  the  Schedule  Success  of  the  program.  (Col  13,  Col  32). 

1 1.  Joint  programs  have  had  significantly  lower  Schedule  Success.  (Col  13,  Col  24). 

12.  There  is  no  correlation  between  M/S  II  assessment  of  risk  and  program  Schedule  Success.  (Col  13,  Col  29). 

13.  Ten  of  twenty- four  programs  had  no  risk  estimate  in  the  DAB  data  analyzed.  (Col  29). 

14.  No  programs  were  rated  high  risk,  (not  permitted  in  EMD).  (Col  29). 

15.  Only  one  program  indicated  Software  Lines  of  Code,  an  accepted  metric  of  S/W  risk.  (Col  28). 

16.  Programs  with  only  one  or  no  Associate  Contractor  had  the  most  Schedule  Success.  (Col  13,  Col  34). 

17. CPAF  contracts  had  the  least  Schedule  Success.  (Col  13,  Col  33). 

D.  WEAPON  SYSTEIM  CATEGORIES 

18.  Airborne  Electronic/EW  systems  had  the  highest  cost  and  schedule  overruns  of  any  group  (Col  6,  10). 

The  ALQ-161  cost  overrun  is  believed  to  be  artificially  constrained. 

Rev  2/ 16/95  Defense  Systems  Managemait  College.  Fort  Bclvoir,  VA  2206Q,  (703)  805-3698 _ _ 
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CHAPTERS 

ANALYSIS 


Primary  Findings 

Table  4  lists  the  primary  eighteen  findings  of  this  study.  The  columnar  notation  shown 
with  each  finding  indicates  the  column  of  the  database.  Appendix  E,  used  to  derive  the 
finding.  Most  findings  are  discussed  in  greater  detail  later  in  this  chapter,  and  in  Chapter 
6.  Also,  several  findings  are  illustrated  in  the  figures  used  herein. 

General 

The  prime  objective  of  the  study  was  to  determine  the  relationship,  if  any,  between  the 
number  of  test  articles  used  in  Engineering  and  Manufacturing  Development  (EMD), 
including  Initial  Operational  Test  and  Evaluation  (lOT&E),  and  the  success  of  that 
program  when  measured  in  cost  and  schedule  overruns  during  EMD.  However,  the 
design  of  the  study  allowed  for  additional  data  to  be  easily  collected,  and  derivative 
findings  identified.  One  derivative,  the  overall  performance  of  the  twenty-four  programs 
is  shown  in  Figure  2  which  indicates  the  results  of  the  planned  and  actual  EMD  cost  and 
schedule  data  for  the  twenty-four  programs.  This  is  the  ratio  of  the  actual  EMD  results  to 
the  planned  figures  for  cost  and  schedule.  A  program  that  came  in  essentially  on  cost  and 
on  schedule,  would  have  a  ratio  of  1:1  for  both  cost  and  schedule.  Eighteen  of  the 
twenty-four  programs  came  in  under  100  percent  cost  and  schedule  overrun,  (ratios  of  2:1 
in  each  area).  For  the  entire  twenty-four  programs,  the  average  cost  overrun  in  the  EMD 
phase  of  the  program  was  45  percent  and  the  average  schedule  overrun  was  63  percent. 

It  was  also  possible  to  separate  program  cost  success  from  program  schedule  success. 
Table  5  lists  the  number  of  success  rankings  at  each  success  level  for  both  cost  and 
schedule.  As  shown,  more  programs  were  better  able  to  control  their  cost  overruns  than 
their  schedule  overruns. 


TABLE  5.  Cost  And  Schedule  Success  Ratings 


Cost 

Schedule 

Program  Success 

Success 

Success 

Rating 

Rating 

Rating 

5.  Very  Successful  (V,S.) 

10 

1 

4.  Successful  (S.) 

4 

8 

3.  Fairly  Successful  (F.S.) 

3 

4 

2.  Marginally  Successful  (M.S.) 

1 

2 

1.  Not  Successful  (N.S.) 

6 

9 

24 

24 
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FIGURE  2.  Actual  vs.  Planned  Schedule  and  Budget 


Low  Rate  Initial  Production  (LRIP) 


Data  and  analysis  relative  to  the  prime  objective  of  the  study  are  shown  in  Figures  3, 4, 
and  5.  Figure  3  is  based  upon  a  sample  of  twenty-one  programs,  since  three  programs  had 
no  Research,  Development,  Test  and  Evaluation  (RDT&E)  funded  LRIP/Initial 
Production  Articles.  One  program  had  procurement  funded  LRIP  articles  only,  no 
RDT«&E  funded  or  full  rate  production  articles.  When  the  test  quantity  is  over  3  percent 
of  the  total  planned  production  quantity,  the  schedule  slippage  historically  did  not  exceed 
50  percent.  Fifty  percent  was  selected  as  the  metric  considering  the  average  schedule 
overrun  of  the  sample  programs,  (63  percent),  and  the  number  of  programs  that  were 
considerably  above  this  figure. 
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FIGURE  3.  LRIP  Test  Articles  and  Schedule  Slippage 
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For  twenty-three  programs,  Figures  4  and  5  show  a  linear  regression  analysis  when 
schedule,  then  cost  overruns,  were  compared  to  the  percentage  of  test  articles  used.  As 
shown,  there  appears  to  be  a  mild  negative  correlation  between  the  percent  of  RDT&E 
funded  test  articles  compared  with  total  production  and  cost/schedule  overruns.  Other 
derivative  data  obtained  from  the  basic  LRIP  test  articles  vs.  success  database  are 
discussed  in  Chapter  6. 

Other  Variables 

The  six  other  variables  shown  in  Table  3  provided  data  amenable  to  graphical 
presentation  in  a  standard  format.  From  the  data  available  in  the  program  Blue  Books  or 
the  Selected  Acquisition  Report  (SAR),  we  assigned  a  success  rating  of  one  to  five  in 
accordance  with  the  plan  outlined  previously.  For  example,  did  the  program  at  Milestone 
(M/S)  II  indicate  it  was  low  risk  or  medium  risk?  Finally,  we  averaged  the  program 
success  ratings  of  low  risk  programs  and  medium  risk  programs,  and  found  in  this 
instance,  medium  risk  programs  had  a  higher  average  program  success  rating  than 
programs  that  were  low  risk. 
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LRiP  Test  Quantity/Total  Production  Percentages 


FIGURE  5.  LRIP  Test  Quantity  vs.  EMD  Budget  Overrun 


Initially  we  assumed  a  low  risk  program  would  have  a  better  probability  of  success  than 
a  higher  risk  program.  The  data  did  not  support  this  assumption.  Programs  that  evaluated 
their  risk  as  low  at  M/S  II  had  an  average  success  rating  of  2.4  as  compared  to  programs 
with  a  medium  risk  evaluation,  that  averaged  a  3.3  success  rating  (see  Figure  6).  The 
asterisks  shown  within  the  figures  are  specific  programs  indicating  the  sample  size  and 
dispersion. 

Whether  or  not  a  program  used  competition  in  the  Demonstration/Validation 
(Dem/Val)  phase,  is  shown  in  Figure  7.  Those  programs  that  used  competition  had  a 
lower  schedule  success  rating  than  programs  that  did  not  use  competition.  The  same 
conclusion  was  reached  for  programs  that  used  competition  in  the  follow-on  EMD  phase, 
(see  Figure  8). 
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Program 
Success 
I  Rating 


M.S.  2 


N.S.  1 


Low  Medium 

{n=10)  {n=4) 

I  Risk  Assessment  at  M/S  111 


FIGURE  6.  Risk  Stated  At  M/S  II 


Programs  that  used 
competition 
had  lower  success 
ratings 


S.  4-1  *** 


M.S.  2 


N.S.  1  J  * 


No  Used 

Competition  Competition 
(n=7)  (n=7) 


FIGURE  7.  Competition  in  DEM/VAL 


Four  EMD  contract  types  were  evaluated  and  the  data  resulted  in  the  following 
program  success  ratings:  Cost  Plus  Incentive  Fee,  Fixed  Price  Incentive,  and  Firm  Fixed 
Price  all  3.3  and  Cost  Plus  Award  Fee  1.0  (see  Figure  9).  The  data  also  showed  that 
programs  using  one  or  no  associate  contractor  (besides  the  Prime)  had  a  success  rating  of 
3.4  compared  to  a  rating  of  2.4  for  the  programs  that  used  more  than  one  associate 
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contractor,  as  depicted  in  Figure  10.  For  probably  the  same  organizational  complexity 
reasons,  single  Service  programs  had  an  average  success  rating  of  3.0  compared  to  Joint 
program’s  success  rating  of  1.8,  seen  in  Figure  11.  Additional  comments  regarding  tliese 
findings  are  included  in  the  next  chapter. 


1 

(2.2) 

M.S.  2"  *******  1 

1 

1 

1 

1 

1 

N.S.  1  -  1 

1 

1 

1 

No 

Competition 

Competition 

Used 

(n=15) 

(n=5) 

FIGURE  8.  Competition  in  EMD 


CPAF  contracts  were  less 
successful 


**  •k'kit 

(3.3)  (3.3)  (3.3) 


M.S.  2- 


N.S.  H  ** 


CPAF  CPIF  FPI  FFP 

{n=2)  (n=3)  (n=4)  (n=3) 


FIGURE  9.  Type  of  Contract  in  EMD 


FIGURE  10.  Number  of  Associate  Contractors  in  EMD 


Program 

Success 

Rating 


M.S.  2  A  * 


N.S.  1  A  **fcicie 


Joint 

Program 

(n=8) 


Single 

Service 

Program 

(n=16) 


FIGURE  11.  Joint  Programs 


CHAPTER  6 

CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 

The  research  described  in  this  report  was  sponsored  by  the  Office  of  the  Director  of 
Operational  Test  and  Evaluation,  Office  of  the  Secretary  of  Defense.  The  contents  of  this 
Technical  Report  do  not  necessarily  reflect  the  opinions  or  policies  of  the  sponsor. 

General 

The  data  presented  herein  were  for  twenty-four  programs,  except  where  otherwise 
noted.  Figure  2  shows  the  cost  schedule  overruns  for  these  programs,  and  Table  4 
(finding  number  7)  shows  the  differences  between  the  overruns  for  twenty-four  programs 
and  eighteen  programs.  This  then,  can  be  considered  the  Statistical  Process  Control  limits 
on  the  current  Department  of  Defense  (DoD)  Acquisition  System.^^  and  this  thought  is 
developed  further  in  reference  22.  The  data  for  twenty-four  programs  also  show  an 
average  Engineering  and  Manufacturing  Development  (EMD)  phase  of  7.4  years.  For 
twenty-three  programs,  the  average  percentage  of  total  Low  Rate  Initial  Production 
(LRIP)  articles  compared  to  total  planned  production  at  Milestone  (M/S)  III,  was  8.7 
percent.  Here  fifteen  programs  had  LRIP  quantities  below  8.7  percent,  and  eight  program 
LRIP  percentages  were  above  the  average.  This  latter  data  may  be  useful  in  studying 
program  concurrency,  a  topic  of  interest  these  days.  Recent  Congressional  action^^ 
requires  at  least  one  LRIP  article  for  a  major  system,  and  seems  to  indicate  LRIP 
quantities  of  up  to  10  percent  of  the  planned  production  quantity  is  acceptable  without 
special  concurrence. 

More  programs  were  better  able  to  control  their  cost  overruns  than  their  schedule 
overruns.  This  is  shown  in  Table  5.  An  assumption  might  be  that  if  a  program  had  a 
considerable  schedule  overrun,  there  would  also  be  a  considerable  cost  overrun,  since 
“time  is  money.”  However,  Figure  2  shows  that  of  the  six  programs  overrun  by  more  than 
100  percent,  only  two  had  comparable  cost  overruns.  The  database  provided  no  factual 
information  on  why  this  occurred.  An  opinion  as  to  why  this  occurred,  is  that  schedule  is 
a  “law  of  physics”  type  of  thing,  where  it  takes  so  long  to  build  a  particular  system  with 
the  performance  specified  in  the  specifications.  Estimated  cost  on  the  other  hand  can  be 
changed  by  several  methods:  de-scoping  the  program  schedule  of  work,  reducing  the 
number  of  planned  test  articles,  delaying  logistical  efforts  until  the  production  phase,  etc., 
can  probably  be  used  to  meet  external  or  Service  generated  program  cost  caps. 

The  data  also  show  that  28  percent  of  the  total  LRIP  quantity  was  acquired  with 
research,  development,  test  and  evaluation  (RDT&E)  funds  and  presumably  used  for 
testing.  The  remaining  72  percent  was  presumably  used  for  other  than  test  purposes. 
Proving  out  the  manufacturing  process  and  ramping  up  the  production  rate  are  the  only 
other  authorized  purposes  of  LRIP  systems.  Since  this  is  accomplished  regardless  of  the 
LRIP  systems  end  use,  a  question  could  be  asked; 
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Is  the  current  28  percent  LRIP  test  usage  and  72  percent  other  than  test 
usage,  the  proper  mix  of  LRIP  systems? 


In  this  regard,  as  mentioned  earlier  (reference  12),  a  recent  Naval  Postgraduate  School 
thesis  concluded  that  the  majority  of  the  problems  which  occurred  during  Initial 
Operational  Test  and  Evaluation/Operational  Evaluation  (lOT&E/OPEVAL)  were  directly 
related  to  test  resource  issues.  It  also  recommended  “sufficient  test  articles  should  be 
produced  and  available  well  before  the  operational  test  is  supposed  to  start. 

LRIP 

Nuances  of  the  LRIP  process  as  used  in  DoD  and  the  design  of  this  research  effort 
produce  uncertainties  that  could  affect  the  level  of  confidence  associated 
with  the  LRIP  vs.  success  results.  Recall  the  primary  determinant  of  the  number  of  LRIP 
test  articles  used  in  the  calculations  was  the  number  purchased  with  RDT&E  funds,  (see 
reference  21).  This  was  compared  to  the  total  planned  production  estimated  at  M/S  III, 
and  the  resultant  percentage  was  used  to  compute  the  probability  of  less  than  a  50  percent 
schedule  slip  by  the  process  previously  described.  Greater  than  three  percent  or  more 
seemed  to  be  the  threshold  of  desired  positive  results. 

However,  the  twenty-four  systems  varied  from  zero  LRIP  articles  purchased  with 
RDT&E  funds  to  464  articles.  It  also  included  total  planned  production  quantities  that 
varied  from  zero,  (one  program  was  canceled)  through  287,  to  a  maximum  of  55,0(X).  It 
depended  entirely  on  the  type  of  system,  and  in  some  cases  very  unique  program 
circumstances.  Therefore,  it  could  be  claimed  that  the  metric,  3  percent  or  more,  might  be 
valid  only  over  a  mid-range  of  planned  production.  The  database  provided  no  factual 
information  to  refute  or  support  this  assumption.  However,  a  review  of  the  actual 
programs  showed  those  using  a  low  number  of  actual  test  articles  were  aircraft  programs 
where  a  few  test  articles  could  be  used  for  many  tests,  thereby  being  equivalent  to  a  larger 
number  of  consumed  test  articles.  Another  group  of  programs  using  a  low  number  of  test 
articles  were,  unfortunately,  electronic  warfare  and  other  programs  that  have  generally 
been  recognized  as  poor  performers  in  EMD  or  canceled  subsequently.  On  the  high  end 
of  actual  RDT&E  funded  test  articles,  there  was  one  ground  tactical  positioning  system, 
and  two  air  launched  tactical  missiles.  The  ground  system  would  require  considerable  test 
articles  for  a  proper  force  on  force,  and  end-to-end  operational  test,  and  the  air  launched 
missiles  were  consumed,  thereby  requiring  more  for  robust  Developmental  Testing  (DT) 
and  Operational  Testing  (OT). 

Another  consideration  is  an  outgrowth  of  additional  guidance  in  the  Comptroller’s 
Regulation  which  states  “Articles...which  are  not  consumed  in  testing  may  be  financed  by 
procurement  or  Operation  and  Maintenance  appropriations.”^^  Therefore,  systems  whose 
test  articles  are  consumed,  such  as  tactical  missiles,  most  probably  were  procured  with 
RDT&E  funds,  and  properly  counted  in  the  study.  Systems  like  tactical  radios,  avionics, 
or  even  aircraft  which  could  be  used  in  testing  and  then  returned  to  operational  use,  may 
have  used  some  test  articles  procured  with  procurement  funds.  In  this  case,  the  number  of 
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LRIP  test  articles  procured  with  RDT&E  funds,  used  in  this  study,  would  be  augmented 
by  additional  quantities  purchased  with  procurement  funds. 

The  net  result  would  be  to  unwittingly  increase  the  number  of  test  articles  relative  to 
total  planned  production,  and  move  the  system’s  percentage  up,  closer  to  the  3  percent 
mark,  or  even  higher  beyond  that  figure.  Again,  the  database  provided  no  factual 
information  to  refute  or  support  this  possibility.  The  study  conclusion  remains  as 
originally  stated,  after  consideration  of  the  nuances  discussed  above,  and  ruminating  on 
their  possible  effects. 

Other  Variables 

The  data  contained  within  the  database  relative  to  risk  estimation  came  as  a  surprise, 
considering  the  amount  of  effort  put  forth  to  indicate  its  importance.  First,  the  data 
contradict  a  reasonable  assumption  that  lower  risk  programs  should  have  a  higher 
probability  of  success.  Second,  the  observed  lack  of  much  risk  estimation  data  within  the 
Blue  Books,  ostensibly  an  important  part  of  the  Defense  Acquisition  Board  (DAB) 
process  represents  lack  of  discipline  in  the  process.  Other  observations  reached  on  the 
five  other  variables  reviewed  in  this  study  stem  directly  from  reduction  of  the  data,  and  are 
considered  self-explanatory. 


Recommendations 

The  database  shown  in  Appendix  E  should  be  made  available  to  other  researchers, 
preferably  by  electronic  transmission. 

This  research  should  continue  into  Phase  II  where  operational  performance  data, 
(effectiveness  and  suitability)  would  be  added  to  the  existing  database.  This  could  be 
accomplished  by  reviewing  the  DOT&E  beyond  LRIP  reports,  and/or  the  Service’s 
Operational  Test  Agency’s  lOT&E  Reports. 

The  existing  database  could  be  expanded.  This  could  be  accomplished  by  reviewing 
some  of  the  other  41  programs  originally  considered,  and/or  by  incorporating  less  than 
major  systems.  The  former  would  result  in  reviewing  programs  somewhat  older  in  time, 
and  the  latter  group  of  programs  may  allow  an  interesting  comparison  between  the  data 
reviewed  at  a  Service  Program  Decision  Review  Meeting,  and  an  Office  of  the  Secretary 
of  Defense  DAB  meeting. 

A  correlation  of  the  seven  program  variables  covered  here  Could  be  made,  using 
multivariate  analysis  techniques  to  determine  the  relative  importance  of  each  variable. 
Additional  research  is  required  to  provide  factual  data  relative  to  the  nuances  of  the  LRIP 
process  and  the  design  of  this  study,  discussed  briefly  above. 
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APPENDIX  A 
DEFINITIONS 

ENGINEERING  DEVELOPMENT  MODEL  -  A  weapon  system  that  is  an  advanced 
prototype  used  during  the  post-MS  II  Engineering  Manufacturing  Development  (EMD) 
phase  to  resolve  design  deficiencies,  demonstrate  maturing  performance,  and  develop 
proposed  production  specifications  and  drawings.  (Source:  TE  Department,  DSMC) 

INITIAL  PRODUCTION  ARTICLE  -  An  initial  production  article  is  a  program  end  item 
system  that  was  produced  in  the  Engineering  and  Manufacturing  Development  (EMD) 
phase  primarily  for  proof  of  design  and  manufacturing  process  testing.  The  article  may  be 
called  Low  Rate  Initial  Production  (LRIP),  or  production  representative  Engineering 
Development  Model  (EDM),  Pre-Production  (PP),  Product  Verification  (PV),  etc.  Either 
RDT&E  or  Production  funds  can  be  used  to  procure  the  article.  However,  the  degree  to 
which  the  initial  production  article  is  production  representative  is  important.  (Source:  R. 
W.  Reig,  DSMC) 

LRIP  TEST  QUANTITY  -  The  number  of  LRIP  articles  purchased  with  RDT&E  funds, 
and  presumably  used  for  all  system  level  testing  in  EMD.  Column  16  of  the  database. 
(Source:  R.  W.  Reig,  DSMC) 

PRODUCTION  REPRESENTATIVE/PRODUCTION  CONHGURATION  -  A  weapon 
system  that  can  be  used  for  Engineering  and  Manufacturing  Development  Phase  (EMD) 
Initial  Operational  Test  and  Evaluation  (lOT&E).  It  is  a  mature  Engineering  Development 
Model  (formerly  referred  to  as  Pre-Production  Prototype)  or  a  Low  Rate  Initial 
Production  (LRIP)  end  item  in  its  final  configuration,  conforming  to  production 
specifications  and  drawings,  and  has  demonstrated  performance  to  appropriate  maturity 
levels.  System  level  Critical  Design  Review,  qualification  testing,  and  Functional 
Configuration  Audit  should  have  been  completed  with  no  significant  deficiencies 
outstanding.  (Source:  TE  Department,  DSMC) 

PROTOTYPE  -  An  original  or  model  on  which  a  later  item  is  formed  or  based.  Usually 
built  during  concept  DEMA^AL  phase  and  tested  prior  to  M/S  II  decision.  (Source: 
Glossary,  DSMC) 
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APPENDIX  B  (CONT) 


Screening  Criteria 

The  study  sample  selected  from  the  population  was  subjected  to  the  following  screening. 

-  All  services  and  some  joint  programs  will  be  considered. 

-  Similar  programs  between  services  will  be  considered  for  comparisons. 

-  Similar  programs  which  used  dissimilar  initial  production  approaches  will 
be  considered. 

-  Programs  associated  with  equipment  used  by  all  Services  will  be 
considered.  Ship  hulls  and  military  satellites  will  not  be  included. 

-  Programs  must  have  completed  M/S  II  before  12/90. 

-  Programs  must  have  completed  M/S  III,  (completed  EMD). 

-  Programs  with  first  operational  delivery  before  1/81  will  not  be  included. 
(Too  old) 

-  Programs  terminated,  in  progress  or  concluded  can  be  considered. 

-  Availability  of  program  data  will  be  an  important  consideration. 

-  All  programs  which  received  BLRIP  reports,  except  satellite/ship 
programs. 
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APPENDIX  C 

INITIAL  LIST  OF  OTHER  VARIABLES  AFFECTING  PROGRAM  SUCCESS 

A.  Decree  of  Program  Concurrency  (Overlap  of  Development  and  Production  Efforts^ 

•  Degree  of  technical  difficulty  identified  at  M/S  II 

•  Modification  vs.  New  Development,  ORD  requirements  vs.  current 
system  capabilities, 

(State  of  the  Art) 

•  Degree  of  software  intensive  development 

•  Total  system  lines  of  code 

•  Program  schedule  vs.  a  nominal  schedule 

•  Months  planned  for  EMD 

•  Degree  the  LRIP  articles  are  production  representative 

•Number  of  Waivers  requested  on  Certification  of  Readiness 
for  lOT&E  or  number  of  ECP’ s 

•  Use  of  competition  during  Dem/Val  phase 

B.  Complexity  of  business,  management  arrangements. 

•  Type  of  contract 

•  Number  of  associate  contractors 

•  Joint  program 

•  Use  of  competition  during  EMD  phase 

C.  Instability  of  Funding  and  Planning. 

•  Initial  Production  Article  phases 

•  Number  originally  planned  vs.  number  actually  used 

•  Joint  program  with  withdrawal  of  one  Service 

D.  Prior  use  of  Prototypes  as  discussed  in  RAND  Report. 

•  Used  or  not  used 
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APPENDIX  D 

PROGRAM  SCHEDULE  AND  COST  SUCCESS 


Proeram 

Schedule 

Success  Ratine 

Cost 

Success  Ratine 

1.  CVHelo 

(months  overrun) 

5 

(%  dollar  overrun) 

5 

2.  MK48 

4 

5 

3.  Avenger 

4 

5 

4.  ACM 

4 

4 

5.  OH-58D 

4 

4 

6.  Trident 

4 

5 

7.  AV-8B 

4 

4 

8.  ASAT 

3 

4 

9.  ALCM 

4 

3 

10.  GPS/U.E. 

1 

5 

11.  LANTIRN 

3 

5 

12.  ATACMS 

4 

5 

13.  Tri-Tac 

2 

1 

14.  HARM 

3 

3 

15.  PLS 

3 

5 

16.  Sgt.York 

2 

1 

17.  MK  50 

1 

2 

18.  FA  ADS 

1 

1 

19.  AMRAAM 

1 

3 

20.  BIB 

1 

5 

21.  JTIDS 

1 

1 

22.  ASPJ 

1 

1 

23.  Maverick 

1 

5 

24.  Tomahawk 

1 

1 
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APPENDIX  E 
Database  Descriptions 


Column 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 


Description 


Study  Population  (Sample),  24  programs. 

The  actual  date  the  program  M/S  II  DAB  meeting  was  held. 

The  planned  date,  (at  M/S  II)  for  the  program  M/S  III  DAB 
meeting. 

The  actual  date  the  program  M/S  III  DAB  meeting  was 
held. 

The  planned  duration  of  phase  II,  (EMD),  in  years. 

The  actual  duration  of  phase  II,  (EMD),  in  years. 

The  ratio  of  actual  duration  of  EMD,  to  the  planned 
duration  (Col  5  divided  by  Col  4). 

The  cost  of  EMD  as  estimated  at  the  M/S  II  DAB  meeting. 

The  FY  dollars  in  which  the  Col  7  figure  is  expressed. 

When  the  Col  8  FY  is  different  than  the  FY  in  which  the 
M/S  II  DAB  meeting  was  held,  the  cost  figure  in  Col  7  has 
been  adjusted  for  inflation  so  that  Column  7  and  Column  8 
are  expressed  in  the  same  FY  dollars. 

The  actual  cost  of  EMD  as  reported  at  the  M/S  III  DAB 
meeting. 

The  ratio  of  the  actual  cost  of  EMD  to  the  planned  cost  of 
cost.  (Col  9  divided  by  Col  7). 

The  program  budget  success  rating  assigned  to  the  program 
in  accordance  with  (iaw)  Table  2. 

The  actual  percentage  the  program  overran  the  planned 
EMD  budget. 

The  program  schedule  success  rating  assigned  to  the 
program  IAW  Table  2. 
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Column 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 


Description 


The  actual  percentage  the  program  overran  the  planned 
EMD  schedule.  (6  minus  1)  xlOO. 

The  actual  months  the  program  overran  the  planned  EMD 
schedule. 

The  number  of  LRIP  systems,  identified  by  several  names, 
(see  Appendix  A  for  definition),  purchased  in  EMD  with 
RDT&E  funds  presumably  used  for  testing,  i.e.,  test 
articles. 

The  number  of  LRIP  systems,  identified  by  several  names 
purchased  in  EMD  with  production  funds,  and  presumably 
used  for  other  than  test  purposes. 

The  total  LRIP  systems  purchased  in  EMD.  (Col  16  plus 
Col  17). 

The  total  planned  production  quantity  as  estimated  at  M/S 
III. 

The  percentage  of  test  articles  when  compared  to  total 
planned  production.  (Col  16  divided  by  Col  19)  x  100. 

The  ratio  of  RDT&E  funded  articles  to  total  EMD  articles 
(Col  16  divided  by  Col  18). 

The  percentage  of  total  LRIP  systems  purchased  in  EMD 
when  compared  to  total  planned  production.  (Col  18 
divided  by  Col  19). 

Taxonomy  of  weapon  system  types,  (see  Database 
Description). 

The  DoD  component,  or  lead  service  developing  the 
system. 

X  Indicates  whether  the  system  is  being  developed  as  a 
Multi-Service  program. 

Yes  indicates  that  the  DOT&E  issued  a  “Beyond  LRIP” 
report  to  Sec  Def,  Congress,  and  the  M/S  III  DAB  meeting 
on  the  date  shown  in  Col  3. 
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Column 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 


Description 


Yes  indicates  the  program  was  on  the  list  of  DOD  SAR 
systems. 

Indicates  whether  the  program  is  a  major  modification,  or  a 
new  development  in  the  EMD  phase. 

Indicates  whether  the  DAB  Blue  Books  recorded  the 
number  of  Software  Lines  of  Code  associated  with  the 
program,  as  a  metric  of  program  risk. 

The  Program  Manager’s  estimate  of  technical  risk  as  stated 
at  M/S  II. 

A  research  study  number  assigned  to  the  program  technical 
risk  (see  Col  29). 

Indicates  whether  the  program  used  prime  contractor 
competition  in  the  Dem/V al  phase  of  the  program. 

Indicates  whether  the  program  used  prime  contractor 
competition  in  the  EMD  phase  of  the  program. 

Indicates  the  type  of  contract  used  in  the  EMD  phase. 
(FFP=Firm  Fixed  Price,  CPIF=Cost  Plus  Incentive  Fee, 
FPI=Fixed  Price  Incentive,  CPAF=Cost  Plus  Award  Fee). 

Number  of  Associate  Contractors  used  at  the  prime  system 
level,  i.e.,  WBS  level  II. 

The  actual  date  of  the  program’s  M/S  I  DAB  meeting. 

The  planned  date  of  the  program  M/S  II  DAB  meeting. 

The  ratio  of  the  actual  duration  of  Dem/Val  to  the  planned 
duration.  (I  minus  35) 

(36  minus  35) 

The  actual  date  of  the  M/S  III  A  DAB  meeting.  This  M/S 
meeting  is  no  longer  used,  but  in  the  past  typically  signified 
the  start  of  the  LRIP  phase  within  the  EMD  program 
phase. 
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The  planned  date  of  the  start  of  the  lOT&E/OPEVAL  test. 

The  actual  date  of  the  start  of  the  lOT&E/OPEVAL  test. 

The  planned  date  to  conclude  the  lOT&E/OPEVAL  test. 

The  actual  date  the  lOT&E/OPEVAL  test  ended. 

The  ratio  of  the  duration  of  the  lOT&E/OPEVAL  test  to 
the  planned  duration.  (Cols  40  and  42  time  interval  divided 
by  Cols  39  and  41  time  interval). 

The  planned  date  of  the  Operational  Delivery  of  the 
system,  generally  the  Initial  Operational  Capability  (IOC) 
date.  (Only  entered  if  unclassified) 

The  actual  date  of  the  Operational  Delivery  of  the  system. 

The  number  of  months  difference  between  the  actual  and 
planned  dates  of  the  system’s  Operational  Delivery. 
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ENDNOTES 


*  The  U.S.  Navy  and  U.S.  Marine  Corps  call  this  specific  phase  of  Operational  Testing  (OT),  Operational 
Evaluation  (OPEVAL).  In  prior  years  the  Office  of  the  Secretary  of  Defense  (OSD)  called  this  particular 
test  the  ‘"’final  dedicated  phase  of  operational  testing  prior  to  Milestone  (M/S)  HI.” 

^  United  States  Code,  Title  10,  Section  2399. 

^  Conversations  between  Dr.  E.  Seglie,  Science  Advisor  to  the  Director  of  Operational  Test  and 
Evaluation  (DOT&E)  and  Professor  R.  Reig,  Defense  Systems  Management  College  (DSMC). 

*  United  States  Code,  Title  10,  Section  2400. 

*  Research  on  Ongoing  Acquisition  Research  (ROAR)  is  a  service  for  subscribers  inside  and  outside 
government  who  are  CURRENTLY  conducting  research  projects  dealing  with  vital  defense  acquisition 
issues. 

®  The  industry  field  trip  is  a  portion  of  the  DSMC  Program  Management  Course.  For  one  week  students 
visit  a  company  that  is  currently  producing  a  major  system.  As  part  of  the  exercise,  the  company  forwards 
to  DSMC  extensive  program  data  similar  to  that  provided  to  DoD. 

’’  Drezner,  J.  A.  (1992).  The  Nature  and  Role  of  Prototyping  in  Weapon  System  Development  (The 
RAND  Corporation,  R-4161-ACQ).  Santa  Monica,  CA. 

*  Drezner,  J.  A.  &  Smith,  G.  K.  (1990).  An  Analysis  of  Weapon  System  Acquisition  Schedules  (The 
RAND  Corporation,  R-3937-ACQ).  Santa  Monica,  CA. 

®  Rich,  M.  &  Dews,  E.  (1986).  Improving  the  Military  Acquisition  Process  (The  RAND  Corporation,  R- 
3373-AF/RC,  p.  6).  Santa  Monica,  CA. 

Drezner,  J.  A.,  et  al.  (1993).  An  Analysis  of  Weapon  System  Cost  Growth  (The  RAND  Corporation, 
MR-29 1-AF,  p.  xiii).  Santa  Monica,  CA. 

“  Gailey,  C.  K.,  Reig,  R.W.,  &  Weber,  W.  (1994).  Director  of  Operation  Test  and  Evaluation  (DOT&E) 
sponsored  research  (current  project)  of  DoD  acquisition. 

Mills,  J.  (1994).  An  Analysis  of  Weapon  System  Readiness  for  Operational  Testing.  Unpublished 
master’s  thesis,  pp.  72  &  74.  The  Naval  Postgraduate  School,  Monterey,  CA. 

Air  Force  Operational  Test  and  Evaluation  Command.  (1991).  TEST  LIMITATIONS.  Experience  of 
the  Air  Force  Operational  Test  and  Evaluation  Center  1974-1990,  p.  15.  Kirtland  AFB,  NM. 

Wiles,  J.A.  (25  February  1994).  Study  Group  Report  on  Evaluation  of  Electronic  System  Acquisition, 
Report  to  the  Under  Secretary  of  Defense  (Acquisition  and  Technology).  Washington,  DC. 

Packard,  D.  (30  June  1986).  A  Quest  for  Excellence.  Final  report  to  the  President  by  the  President’s 
Blue  Ribbon  Commission  on  Defense  Management,  p.  xxiii.  Washington,  DC. 

Mayer,  K.  R.  (1993).  The  Development  of  the  Advanced  Medium  Range  Air  to  Air  Missile:  A  case 
study  of  Risk  and  Reward  in  Weapon  System  Acquisition  (The  RAND  Corporation,  N-3620-AF,  p.vi). 

Glennan,  T.  K.,  Jr.,  Bodilly,  S.  J.,  Camm,  F.,  Mayer,  K.  R.,  &  Webb,  T.  J.  (1993).  Barriers  to 
Managing  Risk  in  Large  Scale  Weapons  System  Development  Programs  (The  RAND  Corporation,  p.  x). 
Santa  Monica,  CA. 

Department  of  Defense,  Office  of  die  Inspector  General.  (9  November  1993).  Low  Rate  Initial 
Production  in  Major  Defense  Acquisition  Programs  (Audit  Report  No.  94-014).  Washington,  DC. 

Some  program  Blue  Books  and  SAR’s  used  constant  dollar  figures  adjusted  to  the  M/S  n  FY  dollars. 
Other  programs  documented  their  costs  in  both  constant  and  then  year  dollars.  Where  time  adjusted  cost 
data  was  not  available,  the  0MB,  CPI  inflation  index  was  used. 

Reig,  R.  (1994).  Insufficiently  Robust  DT&E  Means  Trouble  Ahead  for  OT&E.  Program  Manager,  23 
(4),  12  - 15.  Defense  Systems  Management  College,  Fort  Belvoir,  VA. 

Department  of  Defense,  Financial  Management  Regulation.  (1993).  Vol  2A,  Budget  Formulation  and 
Presentation  (DoD  Comptroller,  DoD  70(K).14-R). 

“Reig,  R.  (1995).  Success  and  Failures  in  the  last  10  years  of  the  DoD  Acquisition  System.  Program 
Manager,  24,  (1),  27-29. 

Public  Law  103-355,  Section  3015,  October  13,  1994. 

Department  of  Defense,  Financial  Management  Regulation.  (1993).  Vol  2A,  Budget  Formulation  and 
Presentation  (DoD  Comptroller,  DoD  7000, 14-R).  Chapter  1,  para  5d,  pp.  1-30. 
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